Event extraction systems and methods

ABSTRACT

Events that are described in either structured data (e.g. HTML web page or email) or text in a natural language description can be extracted and entered into one or more calendars on a user&#39;s device. In one embodiment, selecting an add event command in a calendar application can cause the calendar application to search, without having received any search input, in a database of extracted events, and events extracted within a predetermined period of time can be suggested as events to add to the calendar. In one embodiment, an extracted event can cause a notification to be displayed to a user. Other embodiments are also described herein.

This application claims the benefit of U.S. Provisional PatentApplication No. 62/514,738, filed Jun. 2, 2017, which is incorporatedherein by reference.

BACKGROUND

Users of data processing systems often send messages (e.g., textmessages) or emails about events such as a dinner or lunch at arestaurant or a night at the movies, etc. Moreover, users of dataprocessing systems often use web browsers to make reservations forrestaurants, car rentals, ticketed events (e.g., a baseball game or amovie, etc.), flights, hotels, etc.

Data processing systems, in the past, have been enhanced to includetechniques that recognize different types of data, such as events. See,for example, U.S. Pat. Nos. 7,912,828; 8,423,288; and 8,738,360. Thesetechniques rely upon user interaction to utilize the extracted data.

SUMMARY OF THE DESCRIPTION

Various aspects and embodiments described herein relate to theextraction of events from different types of data. In one embodiment,events can be extracted from natural language description, such ascertain types of emails or text messages or other text content. Inanother embodiment, events can be extracted from structured data such asHTML (e.g., web pages or some types of emails, etc.).

A method which extracts events from natural language descriptions ofevents can include the following operations: extracting an event fromtext having a natural language description; adding the extracted eventto a database containing one or more extracted events; recording datarepresenting a first time associated with the extracted event;displaying a user interface of a calendar application, wherein the userinterface of the calendar application includes an add event command;receiving a selection of the add event command; determining, in responseto the selection, whether the first time which is associated with theevent is within a period of time of a current time; displaying at leasta portion of the extracted event in the calendar user interface if thefirst time is within the period of time of the current time. In oneembodiment, recently extracted events can be suggested as a new event toadd to a calendar in response to a command to add a new event. In oneembodiment, the display of the extracted event can be the result of asearch through an extracted event database, and the results of thesearch display auto completion suggestions of possible extracted eventswhich can be suggested even if no characters are entered into a searchinput field, which can be referred to as a zero word auto completionsuggestion. In other words, the displaying of at least a portion of theextracted event in the calendar user interface can be performed whilezero characters are received in a search input field in the calendaruser interface. When the first time is beyond the period of time of acurrent time, then input characters in the search input field can berequired in order to retrieve the extracted event as a possible searchresult in an auto completion suggestion set of search results. Thenatural language description can be part of a text message or email.

An alternative embodiment can offer auto completion suggestions from asearch of extracted events when a user selects an add event commandwithout regard to a time associated with each extracted event. In thisalternative embodiment, these auto completion suggestions obtained bysearching an extracted event database can be provided without anycharacters being entered into a search input field (a zero word autocompletion suggestion) or after characters have been entered into thefield, which match events in the extracted event database.

In one embodiment, the first time which is associated with an extractedcalendar event can be one of: (a) a time of receipt of a text message oremail; or (b) a time that the extracted event was extracted when thetext message or email was displayed; or (c) a time that the extractedevent was added to the extracted event database.

In one embodiment, the method can further include determining anexpiration date for the extracted event based on data extracted from thenatural language description; and removing the extracted event from thedatabase on or after the expiration date. In one embodiment, thedatabase containing extracted events can include a data structure inwhich the extracted events are ordered by time from most recent to leastrecent. In one embodiment, the method can further include receiving aselection of the displayed extracted event in the calendar userinterface and displaying, in response to the selection of the displayedextracted event, a calendar event creation panel that is pre-populatedwith data from the extracted event to allow editing of an entry based onthe extracted event into a calendar maintained by the calendarapplication. In one embodiment, the extracted events may be placed in asub-calendar, which can be characterized as a “found in application”calendar that is segregated from a user's main calendar. In oneembodiment, the calendar application may support multiple sub-calendars,such as a work calendar, home calendar, etc. as described in U.S.published application US2004/0109025.

Another embodiment in which events are extracted from a natural languagedescription can include the following operations: extracting an eventfrom text having a natural language description; adding the extractedevent to a database containing one or more extracted events; displayinga user interface of a calendar application, the user interface includinga set of one or more dates in a calendar format; receiving a selectionof one of the one or more dates in the calendar format; searching, inresponse to the selection, the database for any extracted events on theselected date, the searching performed while zero characters have beenreceived in a search input field used to receive and cause a search ofthe database; displaying, as one or more candidate events, each of theextracted events in the database on the selected date. In oneembodiment, the method can further include determining an expirationdate for the extracted event based on data extracted from the naturallanguage description; and removing the extracted event from the databaseon or after the expiration date.

Another aspect described herein relates to the extraction of one or moreevents while the user uses a web browser to make a reservation or tootherwise create an event. In one embodiment, a method can include thefollowing operations: receiving a document from a domain; comparing thedomain to a set of domains in a white list of domains; determining, ifthe domain is in the set of domains, whether to continue processing thedocument based upon at least one of a title of the document or a uniformresource identifier (URL) of the document; extracting, from the documentif processing is continued, structured data representing a candidatecalendar event; adding the candidate calendar event to a calendardatabase; and presenting a notification to a user, the notificationshowing at least a portion of data about the candidate calendar event.In one embodiment, the determination of whether to continue processingthe document can be done repeatedly through different web pages of thesame domain (or a domain known to be related to the original domain)through the process of creating the event while using the web browser.In one embodiment, the document is one of a web page or an email or textmessage containing structured data from a business. In one embodiment,the white list is a data structure in memory of the process of the webbrowser, and the data structure is a memory mapped trie and thecomparing is a lookup operation in the memory mapped trie. In oneembodiment, the determination of whether to continue processing can usea machine learned classifier that is trained on manually labeledexamples that are used to learn, based on characterization of title of aweb page or subject line of an email how to classify the structureddata. In one embodiment, the characterization indicates whether theemail or web page is a confirmation of a reservation or an advertisement(a non-event) or a promotion which is not considered an event. In oneembodiment, the calendar database is a private local calendar databasethat is encrypted if stored in a user's private cloud storage account,and the candidate calendar event can be displayed in a sub-calendar thatcan be one of several sub-calendars displayable in a calendarapplication.

In one embodiment, the notification can include at least one of: (a) aclose command that dismisses display of the notification while retainingthe candidate calendar event; or (b) a select command that shows thecandidate calendar event in a user interface of the calendarapplication, wherein the candidate calendar event is editable in thecalendar application; or (c) a delete command that deletes the candidatecalendar event from the calendar database. In one embodiment, thenotification can be a coalesced notification that shows data about a setof candidate events that appear to be related. In one embodiment, themethod can further include removing duplicate events from the calendardatabase based on one or more of: duplicate times; or duplicate title;or duplicate source indicated by one or more domains. In one embodiment,duplicate events can be coalesced into a coalesced notification ratherthan removing them.

Another aspect described herein relates to an architecture for anextraction engine that can use different modules that are shared acrossdifferent categories of events. In one embodiment, an extraction enginecan perform the following operations: receiving a document from adomain, the document containing structured data such as HTML content;determining a language of the document; classifying the document as oneof an event or a non-event; detecting, for an event described by thestructured data, one or more of a location, an address, a date, a time,a phone number, or a uniform resource locator; selecting a dataextractor, from a set of data extractors for different categories ofevents, based on the domain which is determined to be in one of thecategories; invoking, by the selected data extractor, a set of fieldextractors, each of which is configured to extract data from acorresponding type of field in the structured data; extracting, by theset of field extractors, data within the fields of the structured datawherein the flow or order of extracting can be controlled by theselected data extractor; validating extracted data from the fields;generating an output of the extracted data for the event; and adding theevent to a calendar automatically in response to generating the outputif the validating is successful. In one embodiment, the set of dataextractors includes data extractors for one or more of: a restaurantreservation; a car rental reservation; a hotel reservation; a ticketedevent including sports games and shows; a flight reservation; or asocial invitation event. In one embodiment, the method can furtherinclude comparing the domain to a white list of domains; if thecomparison shows that the domain is not in the white list, the methodcan stop before classifying the document. In one embodiment, the methodcan further include pre-processing the document, if the document isclassified as an event, to remove content not associated with the eventprior to extracting data within fields of the structured data. In oneembodiment, the classifying determines whether the document is acancellation of the event. In one embodiment, each data extractor in theset of data extractors controls, when selected, a flow of dataextraction by the field extractors. In one embodiment, the method canfurther include generating a key for the event to compare to other keysto remove duplicate events in the calendar. In one embodiment, eachfield extractor in the set of field extractors extracts candidates fromthe document and scores the candidates for a likelihood of being validdata for a field for the event. In one embodiment, the method can alsoinclude: sending a failure report to a server system if the validatingis not successful, wherein the failure report includes the domain andone or more failure types. The failure reports can be used to comparesuccessful event extractions to failed event extractions and determinewhether or not event extraction software should be updated.

Another aspect described herein relates to the use of extracted eventsin map applications which can display maps and which can be used whentraveling to a location to provide route guidance while traveling to anevent. A method in one embodiment according to this aspect can includethe following operations: extracting an event from structured data ortext having a natural language description, the extracted eventincluding a location; adding the extracted event to a databasecontaining one or more extracted events; receiving a selection todisplay a map application; searching, in response to the selection todisplay the map application, the database for extracted events thatinclude a location; displaying, within the map application, a suggestedoption to show the location of the extracted event. In one embodiment,the method can further include: filtering results from the searching bydetermining whether the location of the extracted event is within athreshold distance of a current location of the data processing systemand if the location of the extracted event is within the thresholddistance of the current location, the extracted event is displayed asthe suggested option. In one embodiment, the method can further include:filtering results from the searching by determining whether a time ofthe extracted event is within a predetermined period of time of thecurrent time, and if the time of the extracted event is within thepredetermined period of time of the current time, the extracted event isdisplayed as the suggested option. In one embodiment, other extractedevents in the database of extracted events are not displayed assuggested options if (a) the other extracted events contain no location,or (b) times of the other extracted events are not within thepredetermined period of time of the current time, or (c) locations ofthe other extracted events are not within the threshold distance of thecurrent location. In one embodiment, the searching can be performedwhile no characters have been entered into a search input field, andhence the suggested options are zero word auto completion suggestions inone embodiment for the map application.

In one embodiment, the database of extracted events has a restrictedaccess requiring an access privilege that the map application or adestination daemon has, while other applications on the data processingsystem do not have the access privilege. In one embodiment, the mapapplication can include a widget accessible from an applicationlaunching user interface. In one embodiment, the location of theextracted event can be validated, and once validated, it can be geocodedto convert an address, such as a street address, to a set of geographiccoordinates such as latitude and longitude. In one embodiment, theextracted event database can include extracted events from one or moresynchronizations with a cloud storage account, used by the dataprocessing system to obtain extracted events from other devices that usethe cloud storage account.

In one embodiment, the method can further include: displaying a command,within the map application's user interface, to add the extracted eventto a calendar that can be displayed through a calendar application;adding the extracted event to the calendar application, and removingduplicate extracted events from at least one of the database or thecalendar application. The method can further include in one embodiment:determining a transportation mode; obtaining, from one or more serversystems, information about traffic or transportation delays; determiningan estimated time of arrival based on the determined transportation modeand the information about traffic or transportation delays; and whereinthe estimated time of arrival is displayed with the suggested optionwithin the map application.

The methods and systems described herein can be implemented by dataprocessing systems, such as one or more smart phones, tablet computers,desktop computers, laptop computers, smart watches, wearable devices,audio accessories, onboard computers, and other data processing systemsand other consumer electronic devices. The methods and systems describedherein can also be implemented by one or more data processing systemswhich execute executable computer program instructions, stored in one ormore non-transitory machine readable media or medium that cause the oneor more data processing systems to perform the one or more methodsdescribed herein when the program instructions are executed. Thus, theembodiments described herein can include methods, data processingsystems, and non-transitory machine readable media such as DRAM memoryand flash memory.

The above summary does not include an exhaustive list of all embodimentsin this disclosure. All systems and methods can be practiced from allsuitable combinations of the various aspects and embodiments summarizedabove, and also those disclosed in the Detailed Description below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a flowchart which illustrates a method for extracting an eventfrom a natural language description.

FIG. 2A shows an example of a user interface of a messaging applicationwhich can include a natural language description of an event.

FIG. 2B shows an example of a user interface of a calendar application.

FIG. 2C shows an example of an auto completion suggestion based upon anextracted event, where the suggestion can be shown in a calendarapplication in response to a selection of a command to add a new eventto the calendar.

FIG. 2D shows an example of a user interface of a calendar applicationwhich can accept search queries that are input into a search inputfield, and the search queries can be used to search extracted events inan extracted event database.

FIG. 2E shows an example of a user interface of a messaging applicationwhich displays a natural language description of a cancellation of anevent.

FIG. 3A shows an example of a user interface of a calendar applicationwhich displays a suggested event which was obtained through a searchthrough an extracted event database.

FIG. 3B shows a method according to one embodiment for displaying one ormore candidate events in response to the selection of a date in acalendar or in response to the selection of a command to add a new eventto the calendar.

FIG. 4 shows an example in block diagram form of an architecture forusing an event extractor with a calendar application.

FIG. 5 is a flowchart which illustrates a method according to oneembodiment in which one or more events can be extracted from structureddata such as a web page.

FIGS. 6A, 6B, and 6C show examples of web pages in a web browser duringa browsing session while a user browses through multiple web pageswithin a domain and a classifier determines, for each such page duringthe browsing session, whether or not to extract an event during thebrowsing session.

FIG. 7 is a block diagram of a software architecture which can be usedto perform the method shown in FIG. 5.

FIGS. 8A, 8B, 8C, and 8D show examples of notifications which can beused, for example, in conjunction with the method shown in FIG. 5.

FIG. 9 is a flowchart which shows a method of using an extraction enginethat uses a hierarchy of components.

FIG. 10 shows an example of an event extraction engine according to oneembodiment.

FIG. 11 shows an example of a hierarchy which can be used to implementthe event extraction engine shown in FIG. 10.

FIGS. 12A, 12B, 12C, 12D, 12E, and 12F show examples of various possiblefields within six different categories of events, which include carrental reservations, ticketed events, restaurant reservations, hotelreservations, flight reservations, and social invitations.

FIGS. 13A, 13B, 13C, 13D, 13E, 13F, and 13G show examples of validationrules which can be used as part of the processing by the eventextraction engine shown in FIG. 10.

FIG. 14A is a flowchart which illustrates a method for improving eventextraction software by monitoring reports of both successful eventextractions and failed event extractions in a large population ofdevices in one embodiment.

FIG. 14B shows a graph which illustrates how the method of FIG. 14A canbe used to trigger an update of event extraction software which then canbe distributed to multiple devices to improve event extraction.

FIG. 15 is a flowchart which illustrates a method in one embodiment forextracting events and using those extracted events to suggest locationsin a map application.

FIG. 16 shows an example of a system which can be used with a mapapplication to suggest locations that are in one or more extractedevents.

FIG. 17 shows an example of a user interface of a map application whichcan display a suggested location that was obtained from a database ofextracted events.

FIG. 18 is a block diagram illustration of an example of a dataprocessing system which can be used with one or more embodimentsdescribed herein.

DETAILED DESCRIPTION

Various embodiments and aspects will be described with reference todetails discussed below, and the accompanying drawings will illustratethe various embodiments. The following description and drawings areillustrative and are not to be construed as limiting. Numerous specificdetails are described to provide a thorough understanding of variousembodiments. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification do not necessarily all refer to the sameembodiment. The processes depicted in the figures that follow areperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software, or a combination of both. Although theprocesses are described below in terms of some sequential operations, itshould be appreciated that some of the operations described may beperformed in a different order. Moreover, some operations may beperformed in parallel rather than sequentially.

One aspect described herein relates to the extraction of one or moreevents from text having a natural language description of the one ormore events. FIG. 1 shows an example of a method which uses eventsextracted from text having a natural language description. The method ofFIG. 1 can use known data extraction techniques which extract data fromnatural language descriptions, where the data includes data for events.Examples for such data extraction techniques are described in U.S. Pat.Nos. 7,912,828 and 8,738,360. These known techniques can be used toperform operation 51 and can be used as part of the event extractor 251shown in FIG. 4. Referring back to FIG. 1, in operation 51, one or moreevents are extracted from text that has a natural language description.FIG. 2A shows an example of a natural language description of an eventdisplayed within a transcript 103 of a user interface of a messagingapplication 101. Referring back to FIG. 1, in operation 53, theextracted event can be added to a database containing one or moreextracted events. Optional operation 55 can then record datarepresenting a first time that is associated with the extracted event.This first time can be used later to determine whether or not to presentthe extracted event as a zero word auto completion suggestion in oneembodiment. In another embodiment, the first time is not used, and theauto completion suggestion which shows extracted events can be displayedwithout regard to time. In operation 57, a user interface of a calendarapplication can be displayed, and this user interface can include acommand to add an event to the calendar. An example of such a command isshown in FIG. 2B which shows a user interface of a calendar application,which includes an add event icon 121; the user can select the add eventicon 121 to cause the calendar application to present a user interface,such as the user interface shown in FIG. 2C to allow the user to add anew event to the calendar. Referring back to FIG. 1, in operation 59,the user interface of the calendar application can receive a selectionof the add event command. The selection may include a touch or tap onthe add event icon or the use of a cursor and mouse to select the addevent icon or the use of other techniques known in the art to select theadd event command represented by the add event icon. Then in operation61 shown in FIG. 1, the system determines, in response to the selectionreceived in operation 59, whether the first time is within a period oftime (e.g., one hour) of the current time; operation 61 is performed inthose embodiments in which the first time is used. If the first time iswithin a period of time of a current time, then the extracted event isdisplayed as a suggested event to add to the calendar, and this is shownas operation 63 in FIG. 1. In one embodiment, the display of theextracted event can occur in response to a search of a database ofextracted events (where no search query has been entered by a user), andall extracted events within a predetermined period of time (e.g., onehour) of the current time can be presented as a suggested event. Inother words, in one embodiment each such suggested event is a zero wordauto completion suggestion that is displayed because the first time ofeach such event is within a period of time of the current time. In oneembodiment, the period of time can be one hour, and all such extractedevents which were extracted within the last hour can be presented as azero word auto completion suggestion of a suggested event (found by asearch engine) in response to the selection of the add event command.

The following example will illustrate the method of FIG. 1 by describinga text message session between two friends. Each friend can use amessaging application to exchange messages with the other friend in aconversation involving lunch. FIG. 2A shows an example on one device ofthat conversation. The user interface 101 of the messaging app includesa transcript 103 which shows three message bubbles 105, 107, and 109.The user interface 101 also includes an input staging area 111 and anon-screen keyboard 112. The text entered into the messaging applicationis analyzed by a hybrid engine leveraging machine learning algorithms(which models were trained to recognize event intents in naturallanguage descriptions) and an event knowledge database. The eventextractor 251 in FIG. 4 is an example of such an event extractor whichanalyzes text in a natural language description such as emails 255 andmessages 253 shown in FIG. 4. The event extractor 251 can then extractevents and store them within the database 257 shown in FIG. 4. Referringback to FIG. 2A, it can be seen that the two friends have agreed to havelunch at a restaurant known as Fred's. In one embodiment, the eventextraction system can assume a default time for lunch, such as 12:30 pm.The event extractor can extract a title of the event, such as “lunch”and can assign a default start time and default duration, if notexplicitly mentioned in the natural language description.

The extracted event, which was extracted from the natural languagedescription, can be added to a database of extracted events, such as thedatabase 257 which can then be searched in response to the selection ofan add event icon, such as the add event icon 121 within the userinterface 120 of the calendar app shown in FIG. 2B. In other words,either user who participated in the messaging conversation shown in FIG.2A can go to their calendar application to bring up the user interface120 and can select the add event icon 121. In response to the selectionof the add event icon, a search of the extracted events database can beautomatically performed to retrieve all recent events within a period oftime, such as one hour of the current time with no search query havingbeen entered by the user. This search can be performed without selectingthe search icon 122 within the user interface 120 of the calendarapplication shown in FIG. 2B. In other words, the search performed inresponse to the selection of the add event icon 121 is automaticallyperformed in response to the selection of the add event icon and doesnot require a separate selection of the search icon 122. The result ofthe search in response to the selection of the add event icon 121 isshown in FIG. 2C, which represents a zero word auto completionsuggestion (from the search by the search engine) of extracted eventswhich have been extracted within the past hour (in this embodiment) ofthe current time. The period of time used in operation 61 in oneembodiment is one hour but could be other periods of time. As shown inFIG. 2C, the extracted event 133 can be an auto completion suggestionreturned as part of the search results where no characters have beenentered into a search input field of the calendar application. Theextracted event 133 shows that the event is considered a candidate eventby the word “maybe” and includes the name of the restaurant and a timeand an invitee who is the other participant in the text messagingsession shown in FIG. 2A. If the user who was operating the messagingapp shown in FIG. 2A has conducted other messaging conversations withinthe past hour, and those other message conversations also createdextracted events, then the auto completion suggestion (from the searchengine 259) of extracted events shown in FIG. 2C would also includethose extracted events because those were also extracted within the pasthour in one embodiment.

The time-based approach shown in FIG. 1 allows a user to retrieve,without entering any characters in a search input field, all recentlyextracted events which have been extracted from recent messages, emails,etc. and other natural language descriptions that have been processed bythe system automatically. Thus, the user need not enter any search queryto retrieve recent extracted events such as those events which have beenextracted in the last hour in one embodiment. It will be appreciatedthat other time periods other than one hour can also be used inalternative embodiments. In one embodiment, the first time which isassociated with the extracted event can be a time of receipt of the textmessage or email that provides the natural language description of theextracted event; alternatively, the first time can be a time that theextracted event was extracted when the text message or email wasdisplayed; alternatively, the first time can be the time that theextracted event was added to the database of extracted events. In eachcase, the first time is compared to the current time to determinewhether it is within a period of time of the current time, such aswithin one hour of the current time. In one embodiment, the eventextractor (such as event extractor 251) can extract an expiration dateor create an expiration date based on rules about expiration and thetext description; for example, a lunch or restaurant reservation can be,by default, treated as expired on the day after the start date, so ifthe start date is Wednesday, August 17th, the expiration date can be, bydefault, Thursday, August 18th. The expiration date, in one embodiment,can be used to remove one or more extracted events that are expired; inone embodiment, all extracted events that are expired can be removedperiodically from the extracted events database so that the searchengine (e.g., search engine 259) does not return stale extracted eventsin the search results. The removal of expired extracted events can alsobe performed in the other embodiments described herein, such asembodiments that extract events from structured data (e.g., HTML contentin web pages or emails in the embodiment shown in FIGS. 7, 10, and 16).

As shown in FIG. 2D, the embodiments described herein can also allow auser to search for extracted events by entering one or more charactersinto the search input field 137 which is displayed within the userinterface 120B of the calendar application. The entry of one or morecharacters into the search input field 137 can cause the search engine259 shown in FIG. 4 to search the database containing extracted events,which is shown as the database 257 in FIG. 4. The results of that searchcan be provided by the search engine 259 to the calendar application 263which can present the user interface as shown in FIGS. 2B, 2C, and 2D.

In one embodiment, the event extractor 251 shown in FIG. 4 can analyzethe natural language description to determine whether an event has beencanceled. This can occur either before the event is added to thedatabase of extracted events, such as database 257, or after the eventsare added to the database of extracted events. The detection of acancellation can cause the event extractor, such as the event extractor251, to remove an extracted event from the database of extracted eventssuch as the database 257 shown in FIG. 4. FIG. 2E shows an example of aconversation containing four message bubbles 145, 147, 149, and 151.Message bubbles 145, 147, and 149 suggest that an event (lunch) has beenscheduled. However, message bubble 151 shows that the event is canceledbecause one of the participants cannot make the event and this can causethe event extractor, such as event extractor 251, to cancel the eventwhich can cause the event to be removed from the database of extractedevents if it has already been added to that database.

FIGS. 3A and 3B show another embodiment in which extracted events can bereturned as search results which can be auto completion suggestionsprovided in the search results. The method shown in FIG. 3B can begin inoperation 201 which extracts an event from text having a naturallanguage description. For example, operation 201 can use the eventextractor 251 to extract an event from a natural language descriptionsuch as descriptions found in messages 253 or emails 255 shown in FIG.4. In operation 203 of FIG. 3B, the extracted events can then be addedto a database containing one or more extracted events. This can be seenin FIG. 4 where the event extractor 251 adds extracted events to thedatabase 257. Referring back to FIG. 3B, a user can select or launch acalendar application which, in operation 205, displays the userinterface of a calendar application which can include various differentways of showing views of a calendar (such as daily view, weekly view,monthly view, etc.). In one embodiment, the view can be a calendarformat which shows one or more dates, such as the format shown in FIG.2B, which shows calendar entries 125, 127, and 129 in a weekly view 123.Then in operation 207 of FIG. 3B, the data processing system can receivea selection of a date in the calendar format that is displayed withinthe user interface of the calendar application. For example, the usercan select the date shown in calendar entry 127 in FIG. 2B. Theselection in operation 207 of FIG. 3B can then cause operation 209 tooccur. In operation 209, the data processing system can search, inresponse to the selection of the date with zero characters entered intoa search input field, the database for any extracted events on theselected date. Referring to FIG. 4, the search engine 259 can search thedatabase 257 for extracted events on the selected date, and the resultsof that search can include one or more extracted events, which can bedisplayed in operation 211 in FIG. 3B as one or more candidate events.FIG. 3A shows an example of a user interface 120C of the calendarapplication which can result from operation 211 in FIG. 3B. In thisexample shown in FIG. 3A, an extracted event 160 has been found by thesearch in operation 209 of FIG. 3B and is displayed in the userinterface 120C. The candidate event is shown as a candidate by the word“maybe” in the extracted event 160 as shown in FIG. 3A. The extractedevent may have been created as a result of an email or text messagebetween the user of the data processing system and another user whichhave exchanged text in a natural language description, which describes alunch at a restaurant known as Mom's Burgers on the date that wasselected in operation 207 of FIG. 3B. If there were multiple extractedevents on the selected date, the calendar user interface, such as userinterface 120C can display multiple extracted events in a scrollablelist. The user of the data processing system which includes the calendarapp can then add the extracted event to the user's calendar by selectingthe add event option 121A or can not add the extracted event to theuser's calendar by selecting the cancel option 156. The user interface120C also allows the user to add a new event by selecting the new eventoption 157, which new event would normally be different than theextracted event 160. In the method of FIG. 3B, the user was not requiredto enter any characters into the search input field (such as searchinput field 137 in the user interface of a calendar application), andhence the search results provided by the search engine can be considereda zero word auto completion suggestion or set of suggestions which arebased upon the selected date which was received in operation 207. Itwill be appreciated that the user could also search for extracted eventsby entering characters into the search input field in the calendarapplication to retrieve calendar events that were manually entered bythe user as well as extracted events in a database of extracted events,such as the database 257. The search engine, such as search engine 259shown in FIG. 4, can then return results to the calendar application 261based upon the characters entered by the user into the search inputfield.

In an alternative embodiment of the method shown in FIG. 3B, operation207 can receive a selection to add a new event rather than a selectionof a date in the calendar application. For example, the user can selectthe add event icon 121 shown in FIG. 2B in this alternative embodimentof operation 207 shown in FIG. 3B. This can cause an alternativeembodiment of operation 209 in which the search engine searches thedatabase of extracted events without regard to date and returns, assearch results, extracted events which are found in the search. Thissearch in the alternative embodiment of operation 209 can be performedwith zero characters having been entered by the user into the searchinput field of the calendar application, and the user can further limitthe search results by entering characters into the search input field toreduce the displayed number of candidate events in the search results.

FIGS. 5, 6A, 6B, 6C, 7, and 8A through 8D show another aspect of theembodiments herein which can be used when processing structured datasuch as web pages that contain HTML content. Embodiments according tothis aspect can use the method shown in FIG. 5 with the architectureshown in FIG. 7, which can ultimately produce a set of one or morenotifications, such as the notifications shown in FIGS. 8A, 8B, 8C, and8D. Referring to FIG. 5, in operation 301, a document, such as a webpage, is received from a domain or other identifier of the source of thedocument. In operation 303, the domain can be compared to a set ofdomains (or other identifiers of sources) in a white list such as thedomain white list 403 shown in FIG. 7. If the domain is not found in thewhite list, method 5 can stop at operation 303 and does not continue aslong as the web browser remains browsing in that domain. In oneembodiment, the white list is a data structure in a memory mapped trie,and the comparison operation performed in operation 303 of FIG. 5 can bea lookup operation in the memory mapped trie. If the domain of thedocument received in operation 301 is found in the white list inoperation 303, then processing continues to operation 305 shown in FIG.5. Operation 305, in one embodiment, can be performed in a looprepeatedly by the classifier 405 shown in FIG. 7 as a user browsesthrough multiple web pages within the same domain, such as the web pages321, 323, and 325 shown respectively in FIGS. 6A, 6B, and 6C. In oneembodiment, the browsing through multiple web pages within the samedomain can occur when a user is booking a registration for a flight or arestaurant or a movie or other event categories, where the bookingprocess requires that the user interact with multiple web pages in thedomain. Referring back to FIG. 5, operation 305 determines whether tocontinue processing the document based upon (in one embodiment) at leastone of a title of the document or a uniform resource identifier, such asa URL (uniform resource locator) of the document. Operation 305 can beseen in FIG. 7, in which the web browser 401 provides the title and URL404 to the classifier 405 which performs operation 305 shown in FIG. 5.As the user browses through multiple web pages within the domain,operation 305 is repeated for each web page, and the web browser 401shown in FIG. 7 repeatedly provides one or more titles and URL 404 tothe classifier 405 which then can indicate whether or not, in decision406, to extract the event in the web page. The decision 406 can indicatewhen to extract based upon the classification of the web page providedby the classifier 405. In a typical example of the loop that operation305 in FIG. 5 can perform, the classifier can return the no decision 406for an initial set of web pages prior to one or more final set of webpages when the classifier provides the yes decision 406 back to the webbrowser 401. This can be seen in FIGS. 6A, 6B, and 6C. In particular,the initial web page 321 is classified as a non-event by the classifier405. Similarly, during the initial stages of the booking process, theclassifier 405 classifies web page 323 as a non-event. This can continuethrough multiple web pages within the domain. At each instance of a newweb page, the web browser 401 in FIG. 7 provides title and URL 404 tothe classifier 405 which then classifies the web page with either a yesdecision (it is an event) or a no decision (it is a non-event, do notextract yet). In the example shown in FIG. 6C, the web page 325 can be aweb page showing that the booking of an event has been confirmed, andthe classifier 405 returns a yes decision 406 to the web browser 401 inFIG. 7, which then provides the HTML content 408 to an event extractor407 shown in FIG. 7. Referring back to FIG. 5, the decision by theclassifier 405 causes processing to continue from operation 305 tooperation 307 in which structured data from the particular web pageclassified as an event by classifier 405 is extracted. The structureddata represents a candidate calendar event. After the structured datahas been extracted in operation 307 by, for example, the event extractor407 in FIG. 7, operation 309 in FIG. 5 can be performed. In operation309, the candidate calendar event can be added to a calendar databasewhich can be a database for a sub-calendar, such as the sub-calendarsdescribed in published U.S. application number 2004/0109025. Forexample, as shown in operation 309 the candidate calendar event can beadded to a calendar which is characterized as a “found in applicationscalendar”. This calendar database can be part of the extracted eventdatabase 409 shown in FIG. 7. The addition of the candidate calendarevent can then cause a notification system, such as notification system411 shown in FIG. 7, to cause operation 311 shown in FIG. 5. Inoperation 311, the data processing system can present a notification toa user, and the notification can show at least a portion of the dataabout the candidate event which was extracted in operation 307 of FIG.5. Referring back to FIG. 7, the notification system 411 can cause thedisplay of notifications, such as the notifications shown in FIGS. 8A,8B, 8C, and 8D, and the user can interact with those notifications tocause the extracted events to be added to one or more calendar databases412 shown in FIG. 7.

In one embodiment, the notifications can be displayed to a usercontemporaneously with the completion of the booking process, and thisis shown in both FIGS. 8A and 8B. In the case of FIG. 8A, the webbrowser window 453 is still displayed on the screen 451 of the user'sdata processing system when the notification 455 is displayed. The webbrowser window 453 can display the final page of the booking processwhich can be a confirmation page providing the details which areextracted by the event extractor 407 shown in FIG. 7. Thus, in oneembodiment, the notification 455 can appear soon after the confirmationweb page (such as web page 325 shown in FIG. 6C) is displayed after theevent extractor has extracted the event and added the extracted event tothe extracted event database, such as database 409 shown in FIG. 7. Inthe case of the user interface shown in FIG. 8B, the web browser window457 is displayed on the screen 456 and multiple notifications 459 aredisplayed at the same time based upon a plurality of extracted eventswhich have been extracted from a booking, such as a car rentalreservation or a flight reservation. FIG. 8C shows an example of thenotification 455 which includes at least a portion of data about thecandidate calendar event in region 465 and also includes a close option467 and a delete option 469. The user can select the region 465 whichcan cause the calendar application to open to allow editing an entry ofthe event into the user's calendar. If the user selects the close option467, the notification is closed without adding the extracted event intothe user's calendar; however, the extracted event can remain in theextracted event database, such as extracted event database 409, untilits expiration date which can be stored in a field as part of the dataabout the particular extracted event. If the user selects the deleteoption 469, the extracted event can be not added to the user's calendarand also can be deleted from the extracted event database, such asextracted event database 409 shown in FIG. 7. FIG. 8D shows an exampleof the user interface of multiple notifications, such as the multiplenotifications 459.

In one embodiment, the classifier 405 shown in FIG. 7 can be a machinelearned classifier that is trained on manually labeled examples of webpages that are used to learn, based on characteristics of a title of aweb page or a subject line of an email whether the email or web page isa confirmation of a reservation or an advertisement or promotion whichis not an event. In one embodiment the events which are extracted by theevent extractor, such as event extractor 407, can be added to a privatelocal calendar database which is separate from the user's main calendar.If the private local calendar database is stored in the user's privatestorage cloud account then it can be encrypted. In one embodiment, theprivate local calendar database that contains extracted events can bedisplayed in a sub-calendar that is one of several sub-calendarsdisplayable in a calendar application. In one embodiment, notificationswhich appear to be for the same event can be coalesced into a singlenotification which can be shown as notifications 459 in FIGS. 8B and 8D.In one embodiment, an identifier of an event, such as a key describedbelow, can be created in order to identify events which are duplicateevents to allow the removal of a duplicate event from the calendar andcalendar database.

Another aspect of the embodiments described herein relates to an eventextraction engine that can use a hierarchy of components, and withinthat hierarchy there can be multiple data extractors with each of themultiple data extractors dedicated to a particular category of eventssuch as hotel reservations, car rental reservations, flightreservations, restaurant reservations, ticketed events, etc. FIGS. 9,10, and 11 show an example of this aspect. Also, the event extractionengine described in this aspect (e.g., event extraction engine 550) canbe used in other aspects, such as in the event extractor 407 and theextraction system 755. In one embodiment, the event extraction engine550 shown in FIG. 10 can perform the method shown in FIG. 9, and theevent extraction engine 550 can use the hierarchy 600 shown in FIG. 11to perform the event extraction process. The method shown in FIG. 9 canbegin in operation 501 in which a document is received from a domain,and the document contains structured data such as HTML content. Thedocument can be, for example, a web page or an email with structureddata or a text message with structured data provided by, for example, anextension app that operates with the messaging application. Referring toFIG. 10, the document 551 is an example of a document received inoperation 501 of FIG. 9. In one embodiment, the method of FIG. 9 canoptionally use a white list of domains such as the white list of domains(or other identifiers of source) 553 shown in FIG. 10. In oneembodiment, the optional use of the white list can be performed prior tooperation 503 of FIG. 9. If the domain is not found in the white list,the method of FIG. 9 can stop at that point and proceed no further forthe current document. On the other hand, if the domain for the receiveddocument is found on the white list, such as white list 553 in FIG. 10,then processing can continue in operation 503. As shown by operation 503in FIG. 9, a classifier can classify the document as either an event ornon-event for purposes of event extraction. In one embodiment, thestructured data within the document can be provided to the machinelearned classifier 557 shown in FIG. 10, which can be similar to theclassifier 405 shown in FIG. 7. In one embodiment, the machine learnedclassifier 557 can be a classifier that is trained manually on labeledexamples that are used to learn how to classify an email or web pagecontaining structured data, and the classification can includeclassifying the structured data as a confirmation of a reservation orother event and can also include classifications of non-events such asadvertisements and promotions, etc. In one embodiment, the machinelearned classifier can learn what the characteristics are of a subjectline for confirmation emails versus an advertising email. For instance,the machine learned classifier can learn when the word “confirmation” ispresent in the subject line or title, this increases the likelihood thatthe email or web page has an event confirmation. Conversely, the machinelearned classifier can learn that when the words “10% off” are presentin the subject line or title of the web page, the email or web page isnot likely to contain an event reservation but is just likely to be acommercial email or advertising that should be discarded. For web pages,the classifier can use the web page's title and full URL forclassification. In one embodiment, the classifier can also usestatistical methods to determine the locale (language and country) ofthe email or web page. The determination of the language can also beperformed by a conventional language identifier 555 shown in FIG. 10which provides an identification of the language in one embodiment tothe machine learned classifier 557 shown in FIG. 10. This can allow themachine learned classifier to use appropriate rules for a particularlanguage in one embodiment. In one embodiment, the machine learnedclassifier 557 can receive the subject line and email headers of emailswhen the document is an email containing structured data. If theclassifier, such as machine learned classifier 557 classifies thedocument as an event, then processing can continue at operation 505 ofFIG. 9. Operation 505 can include detecting from the structured data oneor more of: a location, a date, a time, a phone number, an address, or auniform resource locator or uniform resource identifier. This detectionwhich can occur in operation 505 can be performed by the data detectors561 shown in FIG. 10. In one embodiment, the data detectors 561 can be aset of known data detectors for detecting different types of data, suchas location, date, time, phone number, address, uniform resourcelocator, etc. In one embodiment, each data detector in the set of datadetectors can be dedicated to detecting only one of certain types ofstructured data such as a location, a date, a time, etc. In oneembodiment, the data detectors 561 may perform operation 505 after anoptional preprocessor, such as preprocessor 559, is used to preprocessthe structured data to remove non-event content. This is shown in FIG.10 because the preprocessor 559 precedes the data detectors 561 in thedata flow shown within the event extraction engine 550. In oneembodiment, the preprocessor 559 strips away irrelevant HTMLannotations. For instance, the preprocessor 559 converts “<b> hello</b>” into “hello” and converts “<a href=“http://apple.com”alt=“apple”>” into “<http://apple.com>”, etc.

After detecting data from the structured data of the document inoperation 505, processing can proceed to operation 507 in FIG. 9. Inoperation 507, the event extraction engine can, through a categoryselector such as category selector 563 shown in FIG. 10, select aparticular data extractor from a set of data extractors for thedifferent categories of events. This selection can be made based uponthe domain which is determined to be in one of the categories. Forexample, if the email is received from “OpenTable.com” or the domain ofthe web page is “OpenTable.com” then the category selector 563 wouldselect the data extractor for restaurant reservations, such as the dataextractor for restaurant reservation 609 shown in FIG. 11. In oneembodiment, the category selector 563 shown in FIG. 10 can select one ofthe data extractors shown in FIG. 11 such as data extractors 603, 605,607, and 609. In one embodiment, there can be six such data extractorswhich are selected by the category selector 563 shown in FIG. 10. Thesesix categories are shown in FIGS. 12A, 12B, 12C, 12D, 12E, and 12F. Inone embodiment, the category selector 563 can use a data structure thatincludes a list of domains in each of the six categories. The categoryselector 563 can perform a look-up in the list of the domains anddetermine the category from the category specified for the domain. Forexample, there can be a list of domains for car rental reservations, alist of domains for restaurant reservations, a list of domains forflight reservations, a list of domains for hotel reservations, a list ofdomains for ticketed events, etc. In one embodiment, there is a separatedata extractor for each such category and the set of such dataextractors is shown as the set of data extractors 565 in FIG. 10. Therole of each data extractor is to control the general flow for thecategory. For instance, the restaurant data extractor will attempt toextract the date and time of the event and a location if available, andthen it can call into specialized reusable submodules or fieldextractors which work on a very specific type of extraction within aparticular field. This is shown in operation 509 in which the selecteddata extractor invokes a set of field extractors for each field which isexpected to be extracted for the category of the event. The set of fieldextractors can be field extractors 567 shown in FIG. 10 or the set offield extractors 611 shown in FIG. 11. In one embodiment, for each fieldshown in FIGS. 12A through 12F, there is a separate and distinct fieldextractor for the particular field, although these can be shared acrossthe different categories. For example, a field extractor for the “startdate” field can be shared for both the ticketed events category and therestaurant reservations category and the hotel reservations category andthe flights reservation category, etc.

Referring back to FIG. 9, the set of field extractors which was invokedby the selected data extractor can extract, in operation 511, datawithin the fields of the structured data. The selected data extractordelegates the extraction of all of the necessary properties of an eventin its category and then pieces together these details to create anevent which can be validated in operation 513 shown in FIG. 9 by thevalidator 569 shown in FIG. 10. In one embodiment, the submodules orfield extractors can collect a set of candidates for data extraction inthe document and score them with various criteria in order to retainonly the highest scored item as the extracted data for the particularfield. For example, the start time field extractor can fetch all timecandidates extracted using the data detectors 561 and give higher scoresto candidates that are in close proximity to keywords like “start time”or “your reservation is at” or “begins at”, etc. After the selected dataextractor assembles the extracted event, the data can be validatedusing, for example the validation rules for the particular category (aswell as all categories in the case of FIG. 13A) shown in FIGS. 13B, 13C,13D, 13E, 13F, and 13G. For example, if the category selector, such ascategory selector 563 in FIG. 10 or category selector 601 in FIG. 11selected the category of restaurant reservation, then the validationrules would include the rules shown in FIG. 13A and FIG. 13D in oneembodiment. It will be appreciated that various alternatives exist forthe validation rules, so alternative validation rules could be used inalternative embodiments which may have fewer rules or more rules ordifferent rules in the various categories. Referring back to FIG. 9, inone embodiment, after completing successfully the validation operation(in operation 513) then the event can be added to a calendar databasesuch as a “found in applications” calendar. In one embodiment, theextracted data, prior to adding the extracted event into a database ofextracted events, can be processed to generate a formatted output whichcan be a dictionary representation of the event such as an eventreservation that can comply with, for example, event details using theschema.org web standard for the various categories. In other words, astandard for formatting a particular type of event can be used togenerate a formatted output according to that standard. Moreover theprocess of generating the output in a particular format can also includegenerating a title for the event and metadata about the event such as aduration of the event where the duration is a default duration if theduration is not supplied by the event. In one embodiment, a defaultduration for the event can be inferred when it is not explicit; forexample, the default duration for a restaurant reservation is 1½ hours.The process of generating the output for the extracted event can alsoinclude generating a duplicate key that is used to check whether asimilar event already exists in the user's calendar, and this duplicatekey can be used to de-duplicate the calendar when duplicate events havebeen entered into the calendar. The duplicate key can be, for example,in the case of a restaurant reservation, a key such asrestaurant/reservation identifier/name/time. This key can also be usedwhen a cancellation of the event is extracted so that the cancellationcan be used to remove the canceled event. Once the event is extracted(such as extracted event 571 shown in FIG. 10), it can be added to anextracted event database, such as the extracted event database 573 shownin FIG. 10. In one embodiment, the addition of the extracted event intothe extracted event database can automatically populate the “found inapplications” calendar with the extracted event so that the extractedevent can be found by a search engine which can either be invoked by theuser to search for events within a calendar application or invoked bythe system in response to an event such as the user selecting an addevent command within the calendar application.

As described herein, the event extraction engine 550 can employ ahierarchy of components, such as the hierarchy 600 shown in FIG. 11. Thecategory selector 601 is at the top of the hierarchy and is responsiblefor selecting the particular data extractor for the particular categorythat is selected by the category selector. In one embodiment thecategory selector 601, which can be similar to the category selector 563in FIG. 10, can select the category based upon the domain that was thesource of the document, such as document 551. The category selector 601can then select a particular data extractor which is designed to workwith the selected category. In turn, the selected data extractor, whichcan be one of the data extractors 603, 605, 607, and 609 shown in FIG.11 as well as other data extractors can invoke the appropriate set offield extractors within the set of field extractors 611 depending uponthe particular category. FIGS. 12A, 12B, 12C, 12D, 12E, and 12F show thevarious field extractors for the different categories in one embodiment.The selected data extractor would pick the appropriate field extractorsdepending upon the category to process the data as described herein.

Another aspect of the embodiments described herein is shown in FIGS. 14Aand 14B. Oftentimes, providers of reservation services can change theirweb page formats or their email formats. This may result in the eventextraction engine not being able to successfully extract an eventanymore. These situations can be avoided, in some embodiments, bymonitoring the success and failures of event extractions on many devicesused by users over a period of time. In one embodiment, this can involvetracking the success and failures across all categories of events (suchas the categories shown in FIGS. 12A through 12F) for all providers inall languages. FIG. 14A shows an example of a method which can providethis tracking. In operation 651, reports of successful event extractionscan be received from many client devices, and each report can include anidentifier of the domain (or other source identifier) for a successfullyextracted event. The report can also include a set of metadata about theevent and the document containing the structured data from which theevent was successfully extracted. In one embodiment, the report caninclude a date indicating the date of extraction in addition to anidentifier of the domain. In operation 653, reports of failed eventextractions can be received from many devices on a given date. Eachreport can include an identifier of the domain (or other sourceidentifier) and one or more identifiers of one or more fields for whichextraction failed. These identifiers can be considered an error messagewhich indicates the type of error on a given date for a given domain. Inone embodiment, these metrics can be transmitted by a plurality ofclient devices on a daily basis and can be collected on one or moreserver systems in a privacy preserving way, such as without useridentifiers, etc. and can be used to establish a baseline. The baselinecan be established by, for example, operation 655 which compares thenumber of successful event extractions in a given time period to thenumber of failed event extractions in the same time period and displaysthat comparison over time. In one embodiment, the display can appearlike the graph 670 shown in FIG. 14B which tracks over time successfulevent extractions 673 relative to failed event extractions 671 for thesame domain over the given period of time. The Y axis shown in FIG. 14Bshows the number of reports of event extraction for a particular domain.It can be seen that at time 675 (time T₁) the provider at domain “ABC”probably changed a format in a web page or email which has affected theaccuracy of the event extraction process for one or more fields from theprovider at domain “ABC”. This comparison can reveal a high deviationfrom the baseline indicating that the event extraction engine may needto be modified. In one embodiment this can involve acquiring sampledocuments from the affected domain (for example domain “ABC”). This canbe done, for example, by going to the website of the affected domain andmaking a reservation which is then canceled and collecting sampledocuments such as email confirmation documents and web page confirmationdocuments. These sample documents can then be run through an eventextraction engine to allow the event extraction engine to be modified toallow the sample documents to be properly processed to extract an eventsuccessfully. Once the event extraction software has been modified basedupon this testing, it can be transmitted to the plurality of a user'sclient devices in operation 657.

FIGS. 15, 16, and 17 show another aspect of the embodiments herein whichrelate to automatic searches done in response to, for example, theselection by a user to display a map application. The method shown inFIG. 15 can be used with the system shown in FIG. 16 to produce the userinterface 801 of a map application shown in FIG. 17. Referring now toFIG. 15, in operation 701, an event can be extracted from structureddata or a text having a natural language description, where the eventincludes a location. In operation 703, the extracted event can be addedto a database containing one or more extracted events. Referring now toFIG. 16, the system 750 can receive source materials 753 which can beprocessed by the extraction system 755 to produce extracted events thatare added to the database 757 which contains one or more extractedevents. Hence, the extraction system 755 can process the sourcematerials 753 to perform operation 701 and 703 shown in FIG. 15. In oneembodiment, the source materials can include web pages, emails, textmessages, etc. In one embodiment, the extraction system 755 can besimilar to the event extractor 251 shown in FIG. 4 or similar to theevent extractor 407 shown in FIG. 7 or similar to the event extractionengine 550 shown in FIG. 10. In one embodiment, the extracted event canbe added to a calendar application, such as the calendar application 759shown in FIG. 16. The addition of the extracted event to the calendarapplication can proceed as described herein as either an automaticaddition or an addition after a notification which the user can confirmmanually, or in response to a user's selection of a command, displayedwithin a map application's user interface, to add the extracted event tothe calendar, etc. Referring back to FIG. 15, at some point the user canlaunch or open a map application, and this is shown as operation 705 inwhich the system receives a selection to cause the display of a mapapplication (or a widget of the map application). When the user opens orlaunches a map application, the map application, such as map application751 shown in FIG. 16, can query a destination daemon, such asdestination daemon 761 shown in FIG. 16 to cause a search to beperformed of extracted events in the database of extracted events, suchas the database 757. This search is shown as operation 707 in FIG. 15,and the search can be automatically performed in response to theselection to open or display the map application. In one embodiment, thesearch is limited to or filtered to include only extracted events thatinclude a location which has been validated. In one embodiment, thesearch can be performed even when no search input characters have beenentered into a search input field in the map application. In oneembodiment, the search results can be filtered based on the currentlocation or current time or both parameters. For example, events thatare not within the next eight hours (for flights) and the next fourhours (for any other event found in applications or otherwise containedin the database of extracted events) are disregarded or filtered out bythe search process. In one embodiment, events that don't have astructured geocoded location (such as a latitude and longitude) are alsodisregarded. In one embodiment, events that have a location further awaythan 200 miles of the current location are also disregarded. The currenttime and the current location in one embodiment can be provided, in thesystem 750 shown in FIG. 16, by the clock 763 and the locationdetermination system 767. In one embodiment, the location determinationsystem 767 can include one or more of a GPS system or an assisted GPSsystem or Wi-Fi location techniques, etc. Thus, the automatic searchperformed by operation 707 in FIG. 15 can be filtered based upon thecurrent location and the current time so that only events that are, forexample, within four hours of the current time and within 200 miles ofthe current location are displayed as suggestions in operation 709 inone embodiment. The suggested options can be considered to be autocompletion suggestions based upon no characters having been entered intothe search input field. In one embodiment if there are no extractedevents within a predetermined period of time then there would be nosuggested options which are displayed in operation 709. In oneembodiment, the suggested option which is displayed can include anestimated time of arrival (ETA) based on current traffic conditionsobtained from map or traffic servers, such as the map and trafficservers 765 shown in FIG. 16.

FIG. 17 shows an example of the output of operation 709 of FIG. 15. Theuser interface 801 of the map application shown in FIG. 17 can include amap 803 which, in one embodiment, can show the streets and include anindicator of the location of the suggested option 807. It can be seenthat there are no characters having been entered into the search inputfield 805 and thus the suggested option 807 can be considered to be azero word auto completion suggestion found by the search through theextracted events database, such as the database 757, which search wasperformed in response to the opening or launching of the mapapplication, such as map application 751 which can display the userinterface 801 shown in FIG. 17. The suggested option 807 can include thename of the event and the name of the restaurant in this example as wellas the time and date of the event. In the example shown in FIG. 17, thesuggested option 807 shows that there is a reservation that wasextracted from the source materials (such as source materials 753). Thisreservation is at Mom's Burgers tonight at 7:00 pm. Referring back toFIG. 15, the user in operation 711 can select the suggested option whichcan then cause the display of a map of the location of the extractedevent which is shown in FIG. 17 where X marks the location of Mom'sBurgers. In one embodiment, the events extracted and stored by thesystem shown in FIG. 16 can be any one of the categories shown in FIGS.12A through 12F. In one embodiment the database 757 can enforce arestricted access requiring an access privilege or permission from, forexample, the destination daemon 761, where other daemons and otherapplications will not get access to the database 757 because they do nothave the access privilege or permission controlled by the database 757.In one embodiment, the extraction system 755 in FIG. 16 can include aspart of a validation process, a method for geocoding an address byconverting the address to a set of geographic coordinates. In oneembodiment, the database 757 can include extracted events from one ormore synchronizations with a cloud storage account, used by the systemshown in FIG. 16, to obtain extracted events from other devices that usethe cloud storage account. In one embodiment, the suggested option 807can include an estimated time of arrival which is derived from dataprovided by traffic servers, such as traffic servers 765 shown in FIG.16, where the traffic data is obtained via a network interface such as acellular telephone connection or a Wi-Fi connection. In one embodiment,the map application can display a user selectable command that, whenselected, causes the extracted event to be added to a calendar providedby the calendar application.

The systems and methods described herein can be implemented in a varietyof different data processing systems and devices, includinggeneral-purpose computer systems, special purpose computer systems, or ahybrid of general purpose and special purpose computer systems.Exemplary data processing systems that can use any one of the methodsdescribed herein include server systems, desktop computers, laptopcomputers, tablet computers, smart phones, cellular telephones, personaldigital assistants (PDAs), embedded electronic devices, or otherconsumer electronic devices.

FIG. 18 is a block diagram of data processing system hardware accordingto an embodiment. Note that while FIG. 18 illustrates the variouscomponents of a data processing system that may be incorporated into amobile or handheld device or other electronic device, it is not intendedto represent any particular architecture or manner of interconnectingthe components as such details are not germane to the present invention.It will also be appreciated that other types of data processing systemsthat have fewer components than shown or more components than shown inFIG. 18 can also be used with the present invention.

As shown in FIG. 18, the data processing system includes one or morebuses 1309 that serve to interconnect the various components of thesystem. One or more processors 1303 are coupled to the one or more buses1309 as is known in the art. Memory 1305 may be DRAM or non-volatile RAMor may be flash memory or other types of memory or a combination of suchmemory devices. This memory is coupled to the one or more buses 1309using techniques known in the art. The data processing system can alsoinclude non-volatile memory 1307, which may be a hard disk drive or aflash memory or a magnetic optical drive or magnetic memory or anoptical drive or other types of memory systems (e.g., ROM) that maintaindata even after power is removed from the system. The non-volatilememory 1307 and the memory 1305 are both coupled to the one or morebuses 1309 using known interfaces and connection techniques. A displaycontroller 1322 is coupled to the one or more buses 1309 in order toreceive display data to be displayed on a display device 1323. Thedisplay device 1323 can include an integrated touch input to provide atouch screen. The data processing system can also include one or moreinput/output (I/O) controllers 1315 which provide interfaces for one ormore I/O devices, such as one or more mice, touch screens, touch pads,joysticks, and other input devices including those known in the art andoutput devices (e.g. speakers). The input/output devices 1317 arecoupled through one or more I/O controllers 1315 as is known in the art.

While FIG. 18 shows that the non-volatile memory 1307 and the memory1305 are coupled to the one or more buses directly rather than through anetwork interface, it will be appreciated that the present invention canutilize non-volatile memory that is remote from the system, such as anetwork storage device which is coupled to the data processing systemthrough a network interface such as a modem or Ethernet interface. Thebuses 1309 can be connected to each other through various bridges,controllers and/or adapters as is well known in the art. In oneembodiment the I/O controller 1315 includes one or more of a USB(Universal Serial Bus) adapter for controlling USB peripherals, an IEEE1394 controller for IEEE 1394 compliant peripherals, or a Thunderboltcontroller for controlling Thunderbolt peripherals. In one embodiment,one or more network device(s) 1325 can be coupled to the bus(es) 1309.The network device(s) 1325 can be wired network devices (e.g., Ethernet)or wireless network devices (e.g., Wi-Fi, Bluetooth).

It will be apparent from this description that aspects of the presentinvention may be embodied, at least in part, in software. That is, thetechniques may be carried out in a data processing system in response toits processor executing a sequence of instructions contained in astorage medium, such as a non-transitory machine-readable storage medium(e.g. volatile DRAM or non-volatile flash memory). In variousembodiments, hardwired circuitry may be used in combination withsoftware instructions to implement the present invention. Thus thetechniques are not limited to any specific combination of hardwarecircuitry and software, or to any particular source for the instructionsexecuted by the data processing system. Moreover, it will be understoodthat where mobile or handheld devices are described, the descriptionencompasses mobile devices (e.g., laptop devices, tablet devices),speaker systems with integrated computing capabilities, handheld devices(e.g., smartphones), as well as embedded systems suitable for use inwearable electronic devices.

The present disclosure recognizes that the use of personal informationdata (such as extracted events that can get added to a calendar of auser), in the present technology, can be used to the benefit of users.For example, the personal information data can be used to automaticallypopulate a user's calendar with the extracted event. Further, other usesfor personal information data that benefit the user are alsocontemplated by the present disclosure.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, the presenttechnology can be configured to allow users to select to “opt in” or“opt out” of participation in the collection of personal informationdata during registration for services. In another example, users canselect a partial opt-in in which the data is collected only when certainapplications are used, or the extracted events are added to privateencrypted calendars, etc.

In the foregoing specification, specific exemplary embodiments have beendescribed. It will be evident that various modifications may be made tothose embodiments without departing from the broader spirit and scopeset forth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A non-transitory machine readable medium storingexecutable instructions which when executed by a data processing systemcause the data processing system to perform a method comprising:receiving, while a booking process is being executed within a domain, aplurality of web pages from the domain that are associated with thebooking process; providing, in response to the domain being in a set ofwhitelist domains, a title of each of the plurality of web pages;determining, based upon characteristics of the titles of the pluralityof web pages, whether a web page of the plurality of web pages is aconfirmation page containing a confirmation of a completion of thebooking process; extracting, in response to determining that the webpage contains the confirmation of the completion of the booking process,event data from the web page as a candidate calendar event; adding thecandidate calendar event to a calendar database without adding thecandidate calendar event to a calendar; presenting, contemporaneouslywith the completion of the booking process, the confirmation page and anotification including a selectable region showing at least a portion ofdata about the candidate calendar event, wherein the selectable regionis selectable to initiate a process to add the candidate calendar eventto the calendar.
 2. The medium as in claim 1, wherein the set ofwhitelist domains is a data structure in a memory mapped trie, andfurther comprising comparing the domain to the set of whitelist domainsin a lookup operation in the memory mapped trie.
 3. The medium as inclaim 1, wherein determining whether the web page is the confirmationpage containing the confirmation of the completion of the bookingprocess uses a machine learned classifier that is trained on manuallylabeled examples that are used to learn, based on characteristics of thetitle of the web page, whether the web page is the confirmation page,and wherein the event data representing the candidate calendar event isstructured data.
 4. The medium as in claim 1, wherein the calendardatabase is a private local calendar database that is encrypted ifstored in a user's private cloud storage account, and the candidatecalendar event is displayed in a sub-calendar that is one of severalsub-calendars displayable in a calendar application.
 5. The medium as inclaim 1, wherein the notification includes: (a) a close command thatdismisses display of the notification while retaining the candidatecalendar event in the calendar database without adding the candidatecalendar to the calendar; and (b) a delete command that deletes thecandidate calendar event from the calendar database, and wherein theprocess to add the candidate calendar event to the calendar includesshowing the candidate calendar event in a user interface of a calendarapplication, the candidate calendar event being editable in the calendarapplication.
 6. The medium as in claim 1, wherein the notification is acoalesced notification that shows data about a set of candidate eventsthat appear to be related.
 7. The medium as in claim 1, wherein theportion of data about the candidate calendar event includes one or moreof: a date of event; or a time of event; or a name of event; or alocation of event.
 8. The medium as in claim 1, wherein the methodfurther comprises: removing duplicate events from the calendar databasebased on one or more of: duplicate times; or duplicate titles; orduplicate source indicated by one or more domains.
 9. The medium as inclaim 8, wherein duplicate events are coalesced into a coalescednotification.
 10. The medium as in claim 3, wherein the web page is oneof the plurality of web pages received while the booking process isbeing executed within the domain, and wherein the web page is determinedto contain the confirmation of the completion of the booking processwhen “confirmation” is present in the title of the web page.
 11. Themedium as in claim 1, wherein a web browser receives the plurality ofweb pages and provides the titles of the plurality of web pages to aclassifier, and wherein the classifier determines whether the web pageis the confirmation page.
 12. A method comprising: receiving, while abooking process is being executed within a domain, a plurality of webpages from the domain that are associated with the booking process;providing, in response to the domain being in a set of whitelistdomains, a title of each of the plurality of web pages; determining,based upon characteristics of the titles of the plurality of web pages,whether a web page of the plurality of web pages is a confirmation pagecontaining a confirmation of a completion of the booking process;extracting, in response to determining that the web page contains theconfirmation of the completion of the booking process, event data fromthe web page as a candidate calendar event; adding the candidatecalendar event to a calendar database without adding the candidatecalendar event to a calendar; presenting, contemporaneously with thecompletion of the booking process, the confirmation page and anotification including a selectable region showing at least a portion ofdata about the candidate calendar event, wherein the selectable regionis selectable to initiate a process to add the candidate calendar eventto the calendar.
 13. The method as in claim 12, wherein the set ofwhitelist domains is a data structure in a memory mapped trie, andfurther comprising comparing the domain to the set of whitelist domainsin a lookup operation in the memory mapped trie.
 14. The method as inclaim 12, wherein the classifier is a machine learned classifier that istrained on manually labeled examples that are used to learn, based oncharacteristics of the title of the web page, whether the web page isthe confirmation page, and wherein the event data representing thecandidate calendar event is structured data.
 15. The method as in claim12, wherein the calendar database is a private local calendar databasethat is encrypted if stored in a user's private cloud storage account,and the candidate calendar event is displayed in a sub-calendar that isone of several sub-calendars displayable in a calendar application. 16.The method as in claim 12, wherein the notification includes: (a) aclose command that dismisses display of the notification while retainingthe candidate calendar event in the calendar database without adding thecandidate calendar to the calendar; and (b) a delete command thatdeletes the candidate calendar event from the calendar database, andwherein the process to add the candidate calendar event to the calendarincludes showing the candidate calendar event in a user interface of acalendar application, the candidate calendar event being editable in thecalendar application.
 17. The method as in claim 12, wherein thenotification is a coalesced notification that shows data about a set ofcandidate events that appear to be related.
 18. The method as in claim12, wherein the portion of data about the candidate calendar eventincludes one or more of: a date of event; or a time of event; or a nameof event; or a location of event.
 19. The method as in claim 12, whereinthe method further comprises: removing duplicate events from thecalendar database based on one or more of: duplicate times; or duplicatetitles; or duplicate source indicated by one or more domains.
 20. Themethod as in claim 19, wherein duplicate events are coalesced into acoalesced notification.